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ABSTRACT 



The key to better outcome performance measures for community 
colleges is finding an effective and valid method of determining students' 
educational goals or reasons for attending the institution. In an attempt to 
find a solution to identifying student intent, San Juan College (SJC) 
postulated that student intent at any point in time can be most accurately 
measured by their behavior. The question undertaken in this study is: if we 
look at what courses a student is enrolled in and make certain assumptions 
about intent based on that enrollment, would we have a usable measure of 
student intent? In the preliminary explorations of this question conducted at 
SJC, the emerging answer is "Yes." Student intent data inferred from student 
enrollment does indeed appear to be a stable measure, with face validity, and 
quite usable as a qualifier for performance indicators. SJC identified four 
"enrollment segments" that would be likely educational goals for students 
attending the college: Career, Skill, Transfer, and Lifelong Learner. The 
tables presented in this paper show that reporting performance indicator data 
by enrollment segment provides community colleges an easy method to 
communicate performance data that are appropriate for student intent and the 
multiple missions of the institution. (VWC) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



’ 3 £. cwo?<j 



4 "' ■ 



00 

o 

<N 

VO 

ro 



Q 



W 



U.S DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 

EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

This document has been reproduced as 
\ received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



PERMISSION TO REPRODUCE 

disseminate this material 

BEEN GRANTED BY 



AND 

HAS 



TO J,rL EDUCATIONAL resources 

INFORMATION CENTER (ERIC) 



A New Approach to Including Student Intent Information in 

Performance Measures 



by Dr. Nelle Moore 



A paper presented at the Rocky Mountain Association for Institutional Research annual 
conference, Las Vegas, Nevada, October 1999 



Dr. Nelle Moore 

Director, Institutional Research and Planning 
San Juan College 
4601 College Blvd. 

Farmington, New Mexico 87402 
(505) 599-0438 
n_moore@sjc.cc.nm.us 




2 BEST COPY AVAILABLE' 



4 . 

A New Approach to Including Student Intent Information in Performance Measures 

by Dr. Nelle Moore 

San Juan College, Farmington, New Mexico 

Introduction 

Community Colleges, with their multifaceted missions, face a dilemma when reporting 
data to state agencies for performance measures. Usually these data show community colleges in 
a very poor light because the data do not take into account all of the different reasons that 
students have for attending the college. Performance indicators such as graduation rate, 
persistence, and transfer rate, are appropriate for only some of our students and misrepresent the 
full impact of the college. On the other hand, self-reported student intent data have been so 
notoriously unreliable as to leave no avenue for institutions to make their case. Institutions are 
left in the awkward position of defending their poor performance measures without data to 
substantiate their claims of serving a mixture of educational purposes. 

The key to better performance measures for community colleges is finding an effective and 
valid method of determining students educational goals or reasons for attending the institution. 
Asking the student for their educational goal will always be fraught with error, or subject to 
incomplete data and changes over time. In an attempt to find a solution to identifying student 
intent, San Juan College postulated that student intent at any point in time can be most accurately 
measured by their behavior. The question undertaken in this study is: if we look at what courses a 
student is enrolled in and make certain assumptions about intent based on that enrollment, would 
we have a usable measure of student intent? In the preliminary explorations of this question 
conducted at SJC, the emerging answer is “Yes.” Student intent data inferred from student 
enrollment does indeed appear to be a stable measure, with face validity, and quite usable as a 
qualifier for performance indicators. 

Context for the Study 

For the last two years, the New Mexico Association of Community Colleges, headed by 
Dr. Frank Renz, has been working on developing performance indicators for the two-year 
colleges in anticipation of state legislation. The NMACC wanted to take a proactive stance in 
developing appropriate measures for two-year colleges rather than waiting for the state to dictate 
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measures that would most certainly be based on a four-year model. The 1999 legislative session 
brought the predicted legislation in a bill called “Accountability in Government.” 1 While not 
prescriptive, the bill does call for the identification of performance indicators in higher education 
leading to performance based funding by the year 2004. The bill does not specify what portion of 
funding will be based on performance indicators and leaves it up to each institution to identify its 
“programs” and indicators. However, it also gives oversight authority to the Legislative Finance 
Committee and the Governor’s Budget Committee rather than to the Commission on Higher 
Education. 

The NMACC began its efforts two years ago by using the AACC Core Indicators of 
Effectiveness 2 book as a framework. The NMACC identified 13 indicators to work on initially 
relating just to instructional and fiscal effectiveness and has not yet addressed service to the 
community. Working with the nine independent two-year colleges and the ten two-year branch 
campuses in the state, NMACC asked each institution to select one or more of the indicators to 
develop and pilot test. San Juan College selected persistence rates and transfer rates to develop 
and pilot test appropriate measures for the group. This report is the outcome of that effort. 

Method 

The first step taken by SJC was to identify four “enrollment segments” that would be 
likely educational goals for students attending SJC, given what we knew about our student body: 
Career, Skill, Transfer, and Lifelong Learner. It should be mentioned that different colleges most 
likely have different meaningful segments and certainly will have a different percentage of 
enrollment in each segment than that found at SJC. The enrollment segments and definitions 
should be tailored to match the program mix of your institution. One community college in New 
Mexico provides developmental education courses for all underprepared students at the university 
across the street. They have a significant enrollment of developmental-only students and 
therefore they defined an enrollment segment to fit this category. 

x House Bill 37, as amended, The Legislature of the State of New Mexico, 44 lh Legislature, 
1“ Session, Laws 1999, Chapter 5. 

2 American Association of Community Colleges; Community Colleges: Core Indicators of 
Effectiveness , AACC Communications Services, Washington, D C., 1994. 



Our thinking at SJC evolved this way: let us first identify the enrollment segment that we 
can identify with the highest level of confidence and then move out from there. At SJC we have a 
few programs with highly selective admissions criteria. Courses in these programs are open only 
to majors who have been admitted into those programs, for example nursing and airplane pilot 
training. We can say with great confidence that anyone taking a course in these programs 
“intends” to receive a degree in these fields. Expanding from that base, we said that anyone 
enrolling in vocational fields on a full-time basis most probably intends to receive a degree in 
those fields. We call this enrollment segment “Career.” Furthermore, anyone enrolled in courses 
in vocational fields on a part-time basis may be presumed to be upgrading their job skills. We call 
this segment “Skill.” (Refer to the flow chart on the next page for a visual representation of this 
process.) 

The Career segment can be postulated with fairly strong confidence. The Skill segment, 
however, is less certain. There will certainly be students who intend to get a degree in a 
vocational field but who do not enroll full-time. Let us remember, however, the purpose of this 
exercise. If we want to be able to report performance indicators with greater accuracy and to be 
held accountable as institutions (in performance funding environments) for those measures that 
are most accurate, we want to limit the students in the denominator to those that are most certain 
to be in that category. We know that full-time students have a higher chance of graduating from 
the institution than part-time students. That is not to say that part-time students NEVER 
graduate, but only that if you were to be held accountable for student graduation rates, you would 
want to be accountable for those students who are the most serious, committed to a program, and 
likely to have graduation as a goal than be accountable for those students who are sampling your 
course offerings for their own varied purposes. The point is not to include every student who 
may either now or some time in the future decide they might get a degree. The point of this 
exercise is to REMOVE from the denominator those students whose intent to graduate is 
marginal. 

Having now identified the Career and Skill categories in a hierarchical fashion, we now 
assign students who are not enrolled in any vocational course and who are enrolled full-time to 
the “Transfer” segment. The remaining non-vocational students enrolled part-time are assigned to 
the “Lifelong Learner” segment. Again, is this an accurate assignment in every case? No! 
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Nursing students do transfer on occasion. Students taking general education courses part-time, 
do transfer on occasion. However, in the majority of cases, it is likely that these categorizations 
will capture the general pattern of student intent at SJC and provide more appropriate 
denominators for determining graduation and transfer rates. This segmentation also allows 
institutions the opportunity to develop appropriate performance indicators for the body of casual 
students that benefit from our services rather than confounding our performance indicators by 
analyzing students as if they are one homogeneous whole. Presenting performance data in terms 
of these (or your) enrollment segments properly acknowledges the multiple missions - and 
outcomes - that our institutions achieve. 

Hierarchical Identification of Enrollment Segments 



All Students 



V ocational 

Full-time = CAREER 



V ocational 

Part-time — SKILL. 



Non Vocational 

Full-time ® TRANSFER 



Non Vocational 

Part-time * LIFELONG LEARNER 



Validation 

Stability of Enrollment Segments 

Once the enrollment segments were defined, these definitions were applied to past data to 
see how well they described the enrollment at SJC (Table 1). The first advantage of this 
approach became apparent immediately. Because the segments are identified for each semester, 
they can be applied at any point in time or for a cohort, giving a great deal of flexibility depending 
on the type of indicator that is being developed. We also found that the percent enrollment in 
each segment remained remarkably stable over time. For the first time we saw in numeric terms 
the mix of students that we serve and found a few surprises that have program and marketing 
implications as well as usefulness for performance indicators. 
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Table 1 



Enrollment Segments 




FALL 94 


FALL 95 


FALL 96 


FALL 97 


FALL 98 




U 


% 


# 


% 




% 


# 


% 


# 


% 


Career 


891 


18 % 


971 


19 % 


1,028 


20 % 


1,098 


20 % 


1,233 


22 % 


Skill 


865 


17 % 


829 


16 % 


765 


15 % 


798 


14 % 


741 


13 % 


Transfer 


913 


18 % 


964 


19 % 


914 


18 % 


1,059 


19 % 


1,135 


20 % 


Lifelong Learner 


2,276 


46 % 


2,283 


45 % 


2,407 


47 % 


2,530 


46 % 


2,424 


44 % 


Total 


4,945 




5,047 




5,114 




5,485 




• 5,533 





Telephone Survey of the “Skill” Segment 

Since identifying these four segments, SJC has conducted a telephone survey of students 
that were identified as Skill students in order to determine how well the assumptions we made 
about this segment described the educational intent of the students. The interviewer first 
reminded the students what courses they had recently taken at the college and asked them what 
their purpose was in taking these courses. Fifty-nine percent (59%) of the respondents indicated 
that their purpose was to improve their job skills, which is exactly the purpose that we had 
ascribed to them. Twenty-six percent (26%) of the respondents indicated they took the course(s) 
for personal interest, which crosses into the “Lifelong Learner” segment. Computer courses and 
welding might be examples of courses that could cross over into either segment. Even though 
this method was unable to differentiate between the Skill and Lifelong Learner segments, these 
students were appropriately removed from the Career segment. In terms of outcomes, the Skill 
and Lifelong Learning segment are more like each other than either segment is like the Career 
segment. Therefore, 85% of the respondents were appropriately removed from the Career 
segment. Only 13% of the respondents indicated that they intended to get a degree. 

The findings from the telephone survey show that, while our assumptions are not perfect, 
the Skill category was successful in removing a large percentage of students from the Career 
segment who are not intending to complete a degree. Jhe survey information was an encouraging 
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confirmation of our assumptions about the Skill segment and also led to a further level of 
refinement in the assignment of courses to segment categories. 



r* • 



Persistence Rates 

Our next step was to use the enrollment segments to report persistence rates. The two 
charts presented below indicate face validity of the segments and begin to show the benefit of 
their application. Instead of reporting an overall fall-to-spring persistence rate of 62% for all first- 
time students, and 40% fall-to-fall rate, San Juan College can report that 80% of its transfer 
students persisted fall-to-spring and 46% persisted fall-to-fall while 69% of its career students 
persisted fall-to-spring and 51% persisted fall-to-fall. This is a significant improvement in 
communication of mission and outcomes. 



Persistence Rate 

All First-Time Students 




Fall '94 Fall '95 Fall '96 



HI Fall to Spring ^ Fall to Fall 



Persistence Rate 

By Enrollment Segment 




Career Skill Transfer LifeTeam 



Fall to Spring 



Fall to Fall 



Transfer Rates 

The next application of enrollment segments was to transfer rates. Transfer rates in 
themselves present a difficult dilemma for institutions that do not have state level reporting or 
where, as in New Mexico, questions about FERPA have prevented the state agency from 
reporting transfer data to institutions. Also, for institutions near a state border, large proportions 
of transfers may cross state lines, making student tracking difficult. Furthermore, in keeping with 
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